tests: add markdown link checker #11358

andreizet · 2020-09-01T09:22:42Z

Summary

This is a build related change, it adds a new action that checks for broken links inside markdown files when performing the CI actions.

It will prevent issues like #11089

Related Issues/PRs

googlebot · 2020-09-01T09:22:46Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

andreizet · 2020-09-01T09:39:02Z

@googlebot I signed it!

googlebot · 2020-09-01T09:39:06Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

.github/workflows/ci.yml

connorjclark · 2020-09-02T21:14:22Z

Looks like we are blocked on tcort/markdown-link-check#94

connorjclark · 2020-09-02T21:15:29Z

I like their solution: JabRef/jabref#6695

Would be good to move this to a once-a-week check. Could you change the action to be a cron instead?

andreizet

I like their solution: JabRef/jabref#6695

Would be good to move this to a once-a-week check. Could you change the action to be a cron instead?

Sure, will do.

andreizet · 2020-09-01T09:41:20Z

.github/workflows/ci.yml

@@ -172,3 +172,14 @@ jobs:

    - name: Run smoke tests
      run: yarn smoke --debug -j=1 --retries=2 dbw oopif offline lantern metrics
+


If this is not the place to be, I can move this to a separate .yml file.

andreizet · 2020-09-01T09:42:46Z

.github/workflows/ci.yml

+          use-quiet-mode: 'yes'
+          use-verbose-mode: 'yes'
+          config-file: 'markdown.links.config.json'
+          folder-path: 'docs/'


If there are more files to check, beside the one from docs, we can check them. For all markdown files in the repo, it takes like 11 minutes, and some of them will be marked as 429, since this library has no throttling.

andreizet · 2020-09-01T09:43:10Z

markdown.links.config.json

+{
+  "ignorePatterns": [
+    {
+      "pattern": "^http://www.tmeter.ru"


TMeter seems to be down, we can remove the link or replace TMeter with something else.

andreizet · 2020-09-01T09:44:15Z

markdown.links.config.json

+      "pattern": "^http://www.tmeter.ru"
+    },
+    {
+      "pattern": "https://sites.google.com/a/chromium.org/chromedriver/capabilities#TOC-perfLoggingPrefs-object"


The url is just fine, but markdown-link-check marks it with 4xx response code. I'm gonna check this with them, and after a fix will be available, we can remove this.

andreizet · 2020-09-01T16:53:46Z

docs/readme.md

@@ -133,7 +133,7 @@ $ lighthouse --port=9222 --emulated-form-factor=none --throttling.cpuSlowdownMul

 ## Lighthouse as trace processor

-Lighthouse can be used to analyze trace and performance data collected from other tools (like WebPageTest and ChromeDriver). The `traces` and `devtoolsLogs` artifact items can be provided using a string for the absolute path on disk if they're saved with `.trace.json` and `.devtoolslog.json` file extensions, respectively. The `devtoolsLogs` array is captured from the `Network` and `Page` domains (a la ChromeDriver's [enableNetwork and enablePage options]((https://sites.google.com/a/chromium.org/chromedriver/capabilities#TOC-perfLoggingPrefs-object)).
+Lighthouse can be used to analyze trace and performance data collected from other tools (like WebPageTest and ChromeDriver). The `traces` and `devtoolsLogs` artifact items can be provided using a string for the absolute path on disk if they're saved with `.trace.json` and `.devtoolslog.json` file extensions, respectively. The `devtoolsLogs` array is captured from the `Network` and `Page` domains (a la ChromeDriver's [enableNetwork and enablePage options](https://sites.google.com/a/chromium.org/chromedriver/capabilities#TOC-perfLoggingPrefs-object)).


Seems that a wrongly opened parenthesis was messing up the link. This link was not even transformed by github: https://github.com/GoogleChrome/lighthouse/blob/master/docs/readme.md

nice! this is exactly the kind of thing I wanted to catch.

.github/workflows/ci.yml

andreizet · 2020-09-02T21:23:29Z

Looks like we are blocked on tcort/markdown-link-check#94

markdown-link-check needs an option to retry on 429 codes, but it will not help without this dependency update: link-check

andreizet · 2020-09-04T07:51:09Z

So I've managed to find a solution:
I've splitted the job, so now it checks the files inside /docs, then it checks the ones from root. Also, now it runs every monday at 9 am.

Once the 429 issue will be resolved, I will come back with another pull request, so maybe we can check everything.

https://github.com/andreizet/lighthouse/runs/1070765498?check_suite_focus=true

connorjclark

Since this runs so infrequently, and we've seen it handle all the links, we should be good to merge this without any more changes upstream.

Worse case, we get an email some Mondays with a couple "429s"..

Thanks for taking care of this for us!

This was assigned to @paulirish so I'll defer to him.

connorjclark · 2020-09-08T20:53:52Z

docs/readme.md

@@ -133,7 +133,7 @@ $ lighthouse --port=9222 --emulated-form-factor=none --throttling.cpuSlowdownMul

 ## Lighthouse as trace processor

-Lighthouse can be used to analyze trace and performance data collected from other tools (like WebPageTest and ChromeDriver). The `traces` and `devtoolsLogs` artifact items can be provided using a string for the absolute path on disk if they're saved with `.trace.json` and `.devtoolslog.json` file extensions, respectively. The `devtoolsLogs` array is captured from the `Network` and `Page` domains (a la ChromeDriver's [enableNetwork and enablePage options]((https://sites.google.com/a/chromium.org/chromedriver/capabilities#TOC-perfLoggingPrefs-object)).
+Lighthouse can be used to analyze trace and performance data collected from other tools (like WebPageTest and ChromeDriver). The `traces` and `devtoolsLogs` artifact items can be provided using a string for the absolute path on disk if they're saved with `.trace.json` and `.devtoolslog.json` file extensions, respectively. The `devtoolsLogs` array is captured from the `Network` and `Page` domains (a la ChromeDriver's [enableNetwork and enablePage options](https://sites.google.com/a/chromium.org/chromedriver/capabilities#TOC-perfLoggingPrefs-object)).


nice! this is exactly the kind of thing I wanted to catch.

.github/workflows/check-md-links.yml

paulirish

yeah i also wanted that config file to move out of root. happy that's done. lg!

thanks for working on this @andreizet !

Added markdown links checker

1096aac

andreizet requested a review from a team as a code owner September 1, 2020 09:22

andreizet requested review from paulirish and removed request for a team September 1, 2020 09:22

vercel bot deployed to Preview September 1, 2020 09:22 View deployment

googlebot added the cla: no label Sep 1, 2020

devtools-bot assigned paulirish Sep 1, 2020

devtools-bot added the waiting4reviewer label Sep 1, 2020

andreizet changed the title ~~Added markdown links checker~~ misc(build): added markdown links checker Sep 1, 2020

googlebot added cla: yes and removed cla: no labels Sep 1, 2020

Removed wrongly opened parenthesis from docs/readme.md

d21ba74

vercel bot deployed to Preview September 1, 2020 16:49 View deployment

connorjclark changed the title ~~misc(build): added markdown links checker~~ tests: add markdown-link-check Sep 1, 2020

connorjclark changed the title ~~tests: add markdown-link-check~~ tests: add markdown link checker Sep 1, 2020

connorjclark reviewed Sep 1, 2020

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

Check links in root readme.md

ee91506

vercel bot deployed to Preview September 1, 2020 18:14 View deployment

Ignore the workflow badge when checking urls

ad9a004

vercel bot deployed to Preview September 1, 2020 19:18 View deployment

andreizet commented Sep 2, 2020

View reviewed changes

Moved links checking to a separate yml

78384d4

vercel bot deployed to Preview September 3, 2020 19:57 View deployment

Moved links checking to a separate yml

d9eaa8c

vercel bot deployed to Preview September 3, 2020 20:45 View deployment

Markdown Link Check: Run only on push

15cbf96

vercel bot deployed to Preview September 4, 2020 05:53 View deployment

Markdown Link Check: Run only on push

9bc1a6e

vercel bot deployed to Preview September 4, 2020 05:54 View deployment

Markdown links check: ignore commits & pull requests links

0bf13f2

vercel bot deployed to Preview September 4, 2020 06:22 View deployment

Check only readme.md from root

d033137

vercel bot deployed to Preview September 4, 2020 07:26 View deployment

Check links without sleeping between

5a4b672

vercel bot deployed to Preview September 4, 2020 07:29 View deployment

Markdown links check: added comments

89a16a6

vercel bot deployed to Preview September 4, 2020 07:34 View deployment

Markdown links check: removed on push, added on schedule

5acdd48

vercel bot deployed to Preview September 4, 2020 07:37 View deployment

Markdown links check: removed working links from ignore list

050a827

vercel bot deployed to Preview September 4, 2020 07:39 View deployment

Markdown links check: modified comments

e9391f3

vercel bot deployed to Preview September 4, 2020 07:45 View deployment

connorjclark approved these changes Sep 8, 2020

View reviewed changes

brendankenny reviewed Sep 8, 2020

View reviewed changes

.github/workflows/check-md-links.yml Outdated Show resolved Hide resolved

Moved markdown.links.config.json to .github/workflows/

6d14ca3

vercel bot deployed to Preview September 10, 2020 07:34 View deployment

Removed links checking on push

638c289

vercel bot deployed to Preview September 10, 2020 07:38 View deployment

paulirish approved these changes Sep 13, 2020

View reviewed changes

paulirish merged commit a585105 into GoogleChrome:master Sep 13, 2020

zhangpeidong-peyton pushed a commit to zhangpeidong-peyton/lighthouse that referenced this pull request Sep 14, 2020

tests: add markdown link checker (GoogleChrome#11358)

b38be63

emiliogarza mentioned this pull request Nov 24, 2020

add markdown linting #11345

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: add markdown link checker #11358

tests: add markdown link checker #11358

andreizet commented Sep 1, 2020

googlebot commented Sep 1, 2020

andreizet commented Sep 1, 2020

googlebot commented Sep 1, 2020

connorjclark commented Sep 2, 2020

connorjclark commented Sep 2, 2020

andreizet left a comment

andreizet Sep 1, 2020

andreizet Sep 1, 2020

andreizet Sep 1, 2020

andreizet Sep 1, 2020

andreizet Sep 1, 2020

connorjclark Sep 8, 2020

andreizet commented Sep 2, 2020

andreizet commented Sep 4, 2020 •

edited

Loading

connorjclark left a comment •

edited

Loading

connorjclark Sep 8, 2020

paulirish left a comment

		@@ -172,3 +172,14 @@ jobs:

		- name: Run smoke tests
		run: yarn smoke --debug -j=1 --retries=2 dbw oopif offline lantern metrics

tests: add markdown link checker #11358

tests: add markdown link checker #11358

Conversation

andreizet commented Sep 1, 2020

googlebot commented Sep 1, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

andreizet commented Sep 1, 2020

googlebot commented Sep 1, 2020

connorjclark commented Sep 2, 2020

connorjclark commented Sep 2, 2020

andreizet left a comment

Choose a reason for hiding this comment

andreizet Sep 1, 2020

Choose a reason for hiding this comment

andreizet Sep 1, 2020

Choose a reason for hiding this comment

andreizet Sep 1, 2020

Choose a reason for hiding this comment

andreizet Sep 1, 2020

Choose a reason for hiding this comment

andreizet Sep 1, 2020

Choose a reason for hiding this comment

connorjclark Sep 8, 2020

Choose a reason for hiding this comment

andreizet commented Sep 2, 2020

andreizet commented Sep 4, 2020 • edited Loading

connorjclark left a comment • edited Loading

Choose a reason for hiding this comment

connorjclark Sep 8, 2020

Choose a reason for hiding this comment

paulirish left a comment

Choose a reason for hiding this comment

andreizet commented Sep 4, 2020 •

edited

Loading

connorjclark left a comment •

edited

Loading